Video tripwire

ABSTRACT

A video tripwire system may enable a user to enter a video tripwire by drawing it on a video image or on a snapshot from a video stream. This drawing may be enabled by a graphical user interface.

CROSS-REFERENCE TO RELATED APPLICATION

[0001] This application is a continuation-in-part of co-pending U.S.patent application Ser. No. 09/972,039, filed on Oct. 9, 2001, entitled,“Video Tripwire,” commonly-assigned, and incorporated herein byreference in its entirety.

FIELD OF THE INVENTION

[0002] This invention relates to surveillance systems. Specifically, theinvention relates to video-based surveillance systems.

BACKGROUND OF THE INVENTION

[0003] In its original form, a tripwire was an arrangement in which awire, string, or the like was stretched across a path, and if someone orsomething happened to trip over the wire or otherwise pull it, someresponse was triggered. For example, such a response could be detonatinga landmine, sounding an alarm, or recording an event (e.g., triggering acounter, camera, etc.). Today, tripwires are often, for example,implemented as beams of light (e.g., laser, infrared, or visible); whensomeone or something breaks the beam, a response is triggered.

[0004] An example of a conventional tripwire using a light beam is shownschematically in FIG. 1. A source generates a beam of light, which istransmitted across a path to a receiver. If the beam is broken, then thereceiver no longer receives it. This results in the triggering of someresponse, as discussed above.

[0005] Conventional tripwires are advantageous in that they are at leastconceptually simple to use. They also require a minimum of humanintervention, once they have been installed.

[0006] Conventional tripwires, however, have a number of disadvantages.For example, they can not discriminate between triggering objects ofinterest and those not of interest. As an example, one may be interestedin how many people, but not dogs, walk down a path; however, either aperson or a dog would trigger the tripwire. It is also problematic if agroup of people walk together, resulting in a single triggering of thetripwire, rather than one for each person.

[0007] Furthermore, conventional tripwire arrangements generally involvethe installation of dedicated equipment. For example, considering theexample of a laser tripwire, a laser source and a laser detector must beinstalled across a path of interest. Additionally, such dedicatedequipment may be difficult to install in such a manner that it is noteasily detectable.

[0008] Additionally, a conventional tripwire does not afford a highdegree of flexibility. Conventional tripwires typically detect ifsomeone or something passes across it, only, without regard to directionof crossing. Furthermore, because they extend in straight lines, only,conventional tripwires are limited as to the regions across which theymay be set up.

[0009] Conventional video surveillance systems are also in common usetoday. They are, for example, prevalent in stores, banks, and many otherestablishments. Video surveillance systems generally involve the use ofone or more video cameras, and the video output from the camera orcameras is either recorded for later review or is monitored by a humanobserver, or both. Such a system is depicted in FIG. 2, where a videocamera 1 is trained on the path. Video camera 1 generates video signals,which are transmitted over a communications medium, shown here as acable 2. Cable 2 feeds one or both of a visual display device 3 and arecording device 4.

[0010] In contrast with conventional tripwires, video surveillancesystems can differentiate between people and animals (i.e., betweenobjects of interest and objects not of interest) and can differentiatethe individuals within a group of people walking together. They furtherprovide flexibility over tripwires, in terms of the shape of the regionsthey can monitor. Also, because video surveillance systems are so widelyused, there is no need to install further equipment. However, videosurveillance systems also suffer some drawbacks.

[0011] Perhaps the most significant drawback of conventional videosurveillance systems is that they require a high degree of humanintervention in order to extract information from the video generated.That is, either someone has to be watching the video as it is generated,or someone has to review stored video.

[0012] An example of a prior-art video-based surveillance system can befound in U.S. Pat. Nos. 6,097,429 and 6,091,771 to Seeley et al.(collectively referred to below as “Seeley et al.”). Seeley et al. isdirected to a video security system that includes taking snapshots whenan intrusion is detected. Seeley et al. addresses some of the problemsrelating to false alarms and the need to detect someintrusions/intruders but not others. Image differencing techniques andobject recognition techniques are used in this capacity. However, thereare many differences between Seeley et al. and the present invention, asdescribed below. Among the most severe shortcomings of Seeley et al. isa lack of disclosure as to how detection and recognition are performed.What is disclosed in these areas is in contrast to what is presented inregard to the present invention.

[0013] Another example of a video- and other-sensor-based surveillancesystem is discussed in U.S. Pat. Nos. 5,696,503 and 5,801,943 to Nasburg(collectively referred to below as “Nasburg”). Nasburg deals with thetracking of vehicles using multiple sensors, including video sensors.“Fingerprints” are developed for vehicles to be tracked and are used tosubsequently detect the individual vehicles. While Nasburg does mentionthe concept of a video tripwire, there is no disclosure as to how such avideo tripwire is implemented. Nasburg further differs from the presentinvention in that it is focused exclusively on detecting and trackingvehicles. In contrast, the present invention, as disclosed and claimedbelow, is aimed toward detecting arbitrary moving objects, both rigid(like a vehicle) and non-rigid (like a human).

SUMMARY OF THE INVENTION

[0014] In view of the above, it would be advantageous to have asurveillance system that combines the advantages of tripwires with thoseof video surveillance systems, and this is a goal of the presentinvention.

[0015] The present invention implements a video tripwire system, inwhich a virtual tripwire, of arbitrary shape, is placed in digital videousing computer-based video processing techniques. The virtual tripwireis then monitored, again using computer-based video processingtechniques. As a result of the monitoring, statistics maybe compiled,intrusions detected, events recorded, responses triggered, etc. Forexample, in one embodiment of the invention, the event of a personcrossing a virtual tripwire in one direction may trigger the capture ofa snapshot of that person, for future identification.

[0016] The inventive system may be implemented using existing videoequipment in conjunction with computer equipment. It thus has theadvantage of not requiring extensive installation of monitoringequipment. The inventive system may be embodied, in part, in the form ofa computer-readable medium containing software implementing varioussteps of a corresponding method, or as a computer system, which mayinclude a computer network, executing such software.

[0017] The inventive system may also be used in conjunction with imagingdevices other than conventional video, including heat imaging systems orinfrared cameras.

[0018] One embodiment of the invention comprises a method forimplementing a video tripwire system, comprising steps of: installing asensing device (which may be a video camera or other such device), ifone does not already exist; calibrating the sensing device; establishinga boundary as a virtual tripwire; and gathering data.

[0019] Further objectives and advantages will become apparent from aconsideration of the description, drawings, and examples.

DEFINITIONS

[0020] In describing the invention, the following definitions areapplicable throughout (including above).

[0021] A “computer” refers to any apparatus that is capable of acceptinga structured input, processing the structured input according toprescribed rules, and producing results of the processing as output.Examples of a computer include a computer; a general-purpose computer; asupercomputer; a mainframe; a super mini-computer; a mini-computer; aworkstation; a microcomputer; a server; an interactive television; ahybrid combination of a computer and an interactive television; andapplication-specific hardware to emulate a computer and/or software. Acomputer can have a single processor or multiple processors, which canoperate in parallel and/or not in parallel. A computer also refers totwo or more computers connected together via a network for transmittingor receiving information between the computers. An example of such acomputer includes a distributed computer system for processinginformation via computers linked by a network.

[0022] A “computer-readable medium” refers to any storage device usedfor storing data accessible by a computer. Examples of acomputer-readable medium include a magnetic hard. disk; a floppy disk;an optical disk, like a CD-ROM or a DVD; a magnetic tape; a memory chip;and a carrier wave used to carry computer-readable electronic data, suchas those used in transmitting and receiving e-mail or in accessing anetwork.

[0023] “Software” refers to prescribed rules to operate a computer.Examples of software include software; code segments; instructions;computer programs; and programmed logic.

[0024] A “computer system” refers to a system having a computer, wherethe computer comprises a computer-readable medium embodying software tooperate the computer.

[0025] A “network” refers to a number of computers and associateddevices that are connected by communication facilities. A networkinvolves permanent connections such as cables or temporary connectionssuch as those made through telephone or other communication links.Examples of a network include an internet, such as the Internet; anintranet; a local area network (LAN); a wide area network (WAN); and acombination of networks, such as an internet and an intranet.

[0026] “Video” refers to motion pictures represented in analog and/ordigital form. Examples of video include television, movies, imagesequences from a camera or other observer, and computer-generated imagesequences. These can be obtained from, for example, a live feed, astorage device, an IEEE 1394-based interface, a video digitizer, acomputer graphics engine, or a network connection.

[0027] “Video processing” refers to any manipulation of video,including, for example, compression and editing.

[0028] A “frame” refers to a particular image or other discrete unitwithin a video.

BRIEF DESCRIPTION OF THE DRAWINGS

[0029] The invention is better understood by reading the followingdetailed description with reference to the accompanying figures, inwhich like reference numerals refer to like elements throughout, and inwhich:

[0030]FIG. 1 shows a prior art tripwire system;

[0031]FIG. 2 shows a prior art video surveillance system;

[0032]FIG. 3 shows a video tripwire system according to an embodiment ofthe invention;

[0033]FIG. 4 shows a block diagram of an embodiment of an analysissystem according to an embodiment of the invention;

[0034]FIG. 5 shows a flowchart depicting a method according to anembodiment of the invention;

[0035]FIG. 6 shows a flowchart depicting a first embodiment of thecalibration step shown in FIG. 5;

[0036]FIG. 7 shows a flowchart depicting a second embodiment of thecalibration step shown in FIG. 5;

[0037]FIG. 8 shows a flowchart depicting a third embodiment of thecalibration step shown in FIG. 5;

[0038]FIG. 9 illustrates an exemplary embodiment of the histogrammingstep shown in FIG. 8;

[0039]FIG. 10 shows a flowchart depicting an exemplary embodiment of thesegmenting step shown in FIGS. 7 and 8;

[0040]FIG. 11 shows a flowchart depicting an exemplary embodiment of astep of detecting, a tripwire crossing;

[0041]FIGS. 12 and 13 show “screen shots” depicting exemplaryembodiments of reporting formats; and

[0042]FIG. 14 shows a flowchart depicting an exemplary application ofthe invention.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0043] In describing preferred embodiments of the present inventionillustrated in the drawings, specific terminology is employed for thesake of clarity. However, the invention is not intended to be limited tothe specific terminology so selected. It is to be understood that eachspecific element includes all technical equivalents that operate in asimilar manner to accomplish a similar purpose. Each reference citedhere is incorporated by reference as if each were individuallyincorporated by reference.

[0044] Furthermore, the embodiments discussed below are generallydiscussed in terms of detection of people. However, the invention is notto be understood as being limited to the detection of people. On thecontrary, the video tripwire system in the embodiments discussed belowcan be used to detect objects of all sorts, animate or inanimate.Examples include vehicles, animals, plant growth (e.g., a system thatdetects when it is time to trim hedges), falling objects (e.g., a systemthat detects when a recyclable can is dropped into a garbage chute), andmicroscopic entities (e.g., a system that detects when a microbe haspermeated a cell wall).

[0045]FIG. 3 shows an overview of an embodiment of a video tripwiresystem. As in FIG. 2, sensing device 1 is trained on the path andgenerates output signals. Sensing device 1 may be a video camera, asdiscussed in connection with FIG. 2; however, it may also be any othertype of sensing device that generates a video-type output, for example,a heat-based, sound-based (e.g., sonogram) or infrared-based device. Theoutput of sensing device 1 is transmitted over communication medium 2.Communication medium 2 maybe a cable, for example; however, it may alsobe any other communication medium, for example, RF, a network (e.g., theInternet), or light-wave. If communication over communication medium 2requires modulation, coding, compression, or other communication-relatedsignal processing, means for performing such signal processing areprovided either as part of sensing device 1 or as a separate meanscoupled to sensing device 1 (not shown). Communication medium 2 carriesthe output signals from sensing device 1 to analysis system 5. Analysissystem 5 receives input from and sends output to a user interface 6.User interface 6 may include, for example, a monitor, a mouse, akeyboard, a touch screen, a printer, or other input/output devices.Using user interface 6, a user is able to provide inputs to the system,including those needed to initialize (including creating a virtualtripwire, as will be described below) and to provide commands toanalysis system 5. User interface 6 may also include an alarm or otheralerting device; it may further include or be connected to means forimplementing any other response to a triggering event, for example, asdiscussed above. User interface 6 will also generally include a displaydevice like monitoring device 3 in FIG. 2.

[0046] Analysis system 5 performs analysis tasks, including necessaryprocessing to implement the video tripwire. An embodiment of analysissystem 5 is shown in more detail in FIG. 4. FIG. 4 shows analysis system5 coupled to communication medium 2 and to user interface 6, as alsoshown in FIG. 3. In FIG. 4, analysis system 5 is shown comprisingreceiver 51, computer system 52, and memory 53. Receiver 51 receives theoutput signals of sensing device 1 from communication medium 2. If thesignals have been modulated, encoded, etc., receiver 51 contains meansfor performing demodulation, decoding, etc. Furthermore, if the signalsreceived from communication medium 2 are in analog form, receiver 51includes means for converting the analog signals into digital signalssuitable for processing by computer system 52. Receiver 51 may beimplemented as a separate block, as shown, or it may be integrated intocomputer system 52, in an alternative embodiment. Also, if it is notnecessary to perform any signal processing prior to sending the signalsfrom communication medium 2 to computer system 52, receiver 51 may beomitted entirely.

[0047] Computer system 52 is provided with memory 53, which may beexternal to, as shown, or incorporated into computer system 52, or acombination of both. Memory 53 includes all memory resources required byanalysis system 52 and may also include one or more recording devicesfor storing signals received from communication medium 2.

[0048] In a further embodiment of the invention, sensing device 1 maybeimplemented in the form of more than one sensing device monitoring thesame location. In this case, the data output by each sensing device maybe integrated prior to transmitting data over communication medium 2, orthe outputs of all sensing devices may be transmitted to analysis system5 and dealt with there.

[0049] In yet a further embodiment of the invention, sensing device 1may comprise a number of sensing devices monitoring different locationsand sending their data to a single analysis system 5. In this way, asingle system can be used for surveillance of multiple sites.

[0050] The processes performed by the components shown in FIGS. 3 and 4will become clear in light of the following discussion of the inventivemethods.

[0051]FIG. 5 shows an overview flowchart of an embodiment of theinventive method. If a sensing device 1 has not yet been installed, onemust be installed 71. In many cases, however, such sensing devices mayalready exist. For example, most banks already use video surveillancesystems, so there is no need to install new video cameras. In apreferred embodiment of the system, the sensing device (or devices) isinstalled so as to be stationary. Ideally, it is installed with“natural” orientation (i.e., up in the images corresponding to up in thereal world).

[0052] Once sensing device 1 has been installed, it is necessary tocalibrate it with analysis system 5. System calibration may beperformed, generally speaking, by either explicit calibration, in whichthe system is told (or automatically determines) the necessarycalibration parameters of sensing device 1, or by implicit calibration,in which the system is told (or automatically determines) the size of anobject of interest at various locations in the field-of-view of sensingdevice 1. The purpose of calibration is to provide scale information,i.e., so that the system knows what the size of a person or other objectof interest should be in different image areas. This information isespecially important for the data analysis step 74. Calibration may beperformed in one of, or in a combination of two or more of, three ways:manual numeric calibration, aided segmentation calibration, and fullyautomatic calibration. Flowcharts of embodiments of these methods areshown in FIGS. 6, 7, and 8, respectively.

[0053]FIG. 6 shows a flowchart of an embodiment of the manualcalibration method, which involves explicit calibration, as discussedabove. A user enters parameters 721, via user interface 6, relating tosensing device 1. These parameters may include, for example, focallength of sensing device 1, the height of sensing device 1 from theground, and the angle of sensing device 1, with respect to the ground.The analysis system 5 then generates visual feedback 722; for example,the system may superimpose a person or other object of interest on anactual video frame. The visual feedback is presented to the user at userinterface 6. The visual feedback provides scale information (e.g., thesize of the person or other object of interest relative to itssurroundings), which helps to verify that the calibration is correct.The user then decides if the appearance of the visual feedback isacceptable or if the parameters need to be adjusted 723. If it isacceptable, the process is finished; otherwise, the process loops backfor entry of new parameters 721.

[0054] An embodiment of the aided segmentation calibration method, whichuses implicit calibration and may also involve at least some degree ofexplicit calibration (see below), is shown in FIG. 7. In thisembodiment, a person walks (or some other object of interest moves; thesubsequent discussion will refer to a person but should be understood tobe equally applicable to other types of objects of interest) through thefield of view of sensing device 1 (Step 721A). This is to enable thesystem to determine the expected size of an average person in differentareas of the image. The person walking should be the only moving objectin the field of view during the calibration. The system then segmentsout the moving person 722A. The sizes of the person in different regionsof the image are then used to calibrate (i.e., to determine theparameters as discussed above) 723A. Visual feedback is provided 724A,as in manual calibration, and the user then assesses whether or not theappearance of the image is acceptable 725A. If not, then the user mayadjust the parameters 726A, or, alternatively, the calibration may beentirely redone, with the process looping up to Step 721A (dashedarrow). Which of these options is taken maybe made user selectable. Ifthe appearance is acceptable, on the other hand, then the process iscomplete.

[0055] An embodiment of the fully automatic calibration method, whichinvolves implicit calibration, is shown in FIG. 8. First, information(video information is shown in FIG. 8) is gathered by sensing device 1over an extended period of time, say several hours to a few days 721B.After data has been gathered, objects are then segmented out foranalysis 722B. Histograms are then generated for the various objects invarious regions of the image 723B. Details of this step are furtherillustrated in FIG. 9.

[0056]FIG. 9 shows histogramming step 723B embodied as a two-stepprocess, although the invention is not limited to this process. In Step1, the system determines “insalient” image regions, i.e., regions inwhich there are too many confusing objects to track objects reliably. Asa result, only objects that can be tracked with high confidence areused; in one embodiment of the invention, these are the only objectsthat are stored. In Step 2, the system uses only the remaining imageregions and forms histograms of objects detected in those regions. Asindicated in Step 2, and as shown in FIG. 8, the system then uses thehistograms to determine the average size of a person in each region ofthe image 724B. This information is then used to calibrate the system725B. This latter process may be implemented similarly to step 723A ofFIG. 7.

[0057] The step of determining the average size of a person in an imageregion 724B is carried out only if a sufficient number of objects toresult in a meaningful determination are logged in a given region. Thenumber of determinations needed for a meaningful histogram may bedetermined empirically and may depend, for example, on the amount andtype of activity to which the tripwire will be exposed. For suchregions, peaks are detected in the histograms. The highest peak in eachimage region, i.e., the most frequent occurrence, is assumed to be asingle persons of this information is determined, then calibration issuccessfully carried out 725B, and the system is able to signal itsreadiness for actual operation.

[0058] The process of FIG. 8 is typically carried out without humanintervention. However, it is possible for a user to provide time windowsduring which most objects are expected to be individual persons, toreduce the problem of trying to differentiate groups of people. Suchtime windows may be imposed either in the step of obtaining information721B or in further processing steps.

[0059] Each of the automated calibration methods (aided and fullyautomatic) requires the segmentation of images into foreground objectsand background (see steps 722A and 722B in FIGS. 7 and 8, respectively).An embodiment of this process is illustrated in FIG. 10. The exemplaryembodiment consists of three steps: pixel-level background modeling7221; foreground detection and tracking 7222; and object analysis 7223.

[0060] The objective of pixel-level background modeling 7221 is tomaintain an accurate representation of the image background and todifferentiate background (BG) pixels from foreground (FG) pixels. In anexemplary embodiment, this step implements the process disclosed incommonly-assigned U.S. patent application Ser. No. 09/815,385, entitled,“Video Segmentation Using Statistical Pixel Modeling,” filed Mar. 23,2001, and incorporated herein by reference in its entirety. The generalidea of the exemplary method is that a history of all pixels ismaintained over several frames, including pixel values and theirstatistics. A stable, unchanging pixel is treated as BG. If thestatistics of a pixel change significantly, it will be considered to beFG. If the pixel stabilizes again, it will revert to being considered aBG pixel. This method serves to alleviate sensor noise and toautomatically address changes to the background (for example, in astore, when a person removes an item from a shelf, the shelf willinstantaneously be treated as FG but will revert to BG after the scenere-stabilizes).

[0061] The objective of foreground detection and tracking 7222 is tocombine the FG pixels into FG objects and to track them over a number offrames, to guarantee spatio-temporal consistency. This obtains sets ofpixels determined to be FG pixels, as well as their statisticalproperties, from the pixel-level background modeling 7221. In anexemplary embodiment, the FG pixels are spatially merged into larger FGobjects using simple morphology and connected component detection,techniques that are well-known in the art. These objects are trackedusing correlation methods over several frames to obtain reliable sizeinformation. Exemplary tracking techniques are discussed in, forexample, commonly-assigned co-pending U.S. patent application Ser. No.09/694,712, entitled, “Interactive Video Manipulation,” filed Oct.24,2000, and incorporated herein by reference in its entirety. See,also, e.g., Wren, C. R. et al., “Pfinder: Real-Time Tracking of theHuman Body,” IEEE Trans. on Pattern Matching and Machine Intelligence,Vol. 19, pp. 780-784,1997; Grimson, W. E. L. et al., “Using AdaptiveTracking to Classify and Monitor Activities in a Site,” CVPR, pp. 22-29,June 1998; and Olson, T. J. and Brill, F. Z., “Moving Object Detectionand Event Recognition Algorithm for Smart Cameras, IUW, pp. 159-175, May1997. Each of these references is to be considered as being incorporatedby reference herein in its entirety.

[0062] The third step, object analysis 7223, has a number of functions.Object analysis 7223 may serve to separate and count objects; todiscriminate between objects of interest (e.g., people) and “confusers”(e.g., shopping carts); to determine an object's direction of motion;and to account for occlusions of objects. In an illustrative embodiment,determinations regarding an object are made based on one or more of: itssize; its internal motion; the number of head-like protrusions (e.g., ifpeople are the objects of interest); and face detection (for example,again, in the case in which people are the objects of interest).Techniques for performing such functions are known in the art, andexamples of such techniques are discussed in, for example, Allmen, M.,and Dyer, C., “Long-range Spatiotemporal Motion Understanding UsingSpatiotemporal Flow Curves,” Proc. IEEE CVPR, Lahaina, Maui, Hi., pp.303-309, 1991; Gavrila, D. M., “The Visual Analysis of Human Movement: ASurvey,” CVIU, Vol. 73, No. 1, pp. 82-98, January 1999; Collins, Lipton,et al., “A System for Video Surveillance and Monitoring: VSAM FinalReport,” Robotics Institute, Carnegie-Mellon University, Tech. Rept. No.CMU-RI-TR-00-12, ; May 2000; Lipton, A. J., et al., “Moving TargetClassification and Tracking from Real-Time Video,” 1998 DARPA IUW, Nov.20-23, 1998; and Haering, N., et al., “Visual Event Detection,” VideoComputing Series, M. Shah, Ed., 2001. Each of these references is to beconsidered as being incorporated by reference herein in its entirety.

[0063] Returning now to FIG. 5, the step of calibration 72 is followedby a step of initializing the system 73. This step permits a user toenter various parameters relating to how the system will gather, respondto, and report data. First, a user may superimpose one or more lines ofinterest on the image; these lines will serve as one or more tripwires.The lines may be of any orientation and may be placed almost anywhere inthe image; the exception is that the lines may not occur too close toimage boundaries because the object (e.g., person) crossing the linemust be at least partially visible on both sides of the line fordetection to occur. In an illustrative embodiment, it is assumed thatthe tripwire is on the ground in the image; that is, detection occurswhen the bottom portion of an object (e.g., a person's legs) crosses theline. In a more general embodiment, the user may set a height above theground for each line.

[0064] Other parameters that may be initialized include a time intervalof active detection; a direction of crossing each line as a criterionfor event detection (for example, to determine when a person enters anarea, as opposed to when it is desired to determine when a person eitherenters or exits the area); and sensitivity of the detection.

[0065] Embodiments of this invention may include various different typesof tripwires. For example, a video tripwire need not be straight; one ormore curved tripwires may be drawn that follow the contour of one ormore regions in a scene. In a similar vein, a video tripwire need not bea single linear segment; a video tripwire may comprise a multi-segmenttripwire that is made up of more than one linear segment. Furthermore, avideo tripwire need not merely comprise a single tripwire; on thecontrary, a video tripwire may comprise “multiple” parallel tripwiresthat may, for example, require an object to cross all of the tripwiresin a particular order or within a particular period of time. Othervariations may be possible, as well, and the invention is not limited tothese examples.

[0066] Embodiments of this invention may include a graphical userinterface (GUI). In such embodiments, the user may initialize the systemby literally drawing a tripwire on a video image, or an image that is asnapshot from a video stream (e.g., such a “snapshot” may be a frame ofa video stream or may be separately acquired). This may be done using a“point and click” interface, wherein a user may select a point on animage using a pointing device, such as a mouse, and then drag a tripwirealong the image, thus designating the tripwire. Other components of atripwire rule, such as directionality (left-to-right, right-to-left,either), object type (human, vehicle, animal, etc.), object speed, etc.,may also be selected using a “point-and-click” interface. For example,directionality may be selected as options on a graphical menu selectedusing, for example, a pointing device, such as a mouse; object type maybe selected from a list or pull-down menu using, for example, a pointingdevice, such as a mouse; and so on.

[0067] Another function of initialization 73 is for the user to selectvarious logging options. These options determine what data is collectedand may include, but are not limited to:

[0068] logging only when people (or, in general, objects of interest)cross;

[0069] logging only when two or more people cross;

[0070] logging all crossings;

[0071] logging only when there is a high degree of confidence in thedetection of a crossing;

[0072] logging only detection statistics;

[0073] taking a “snapshot” or creating a whole video around a detectionevent.

[0074] By a “snapshot,” it is meant that a still image is created, whichmay simply be a particular video (or other sensing device) frame, orwhich may be independently generated.

[0075] These various options, in combination, may be considered a videoevent rule. A video event rule may comprise a prescribed action (such asa “human” crossing a “virtual tripwire” in a prescribed direction) and aprescribed response (such as logging the alert with text and video to adatabase and sending an e-mail to a particular email address). Videoevent rules may encompass more complex activities involving othervirtual video features, such as areas of interest, along with otherclasses of activities, such as loitering, leaving a bag behind, orstealing an item, and other types of response, such as activating aDigital Video Recorder (DVR) or sounding an audible alarm.

[0076] After initialization 73, the system operates to collect andanalyze data 74. If the user has entered a time window, the systemstarts processing when it is within this time window. When it detects atripwire event (of a particular type, if specified by the user), it islogged along with accompanying information; types of accompanyinginformation will become apparent below in the discussion of datareporting. In the context of some applications, a tripwire event maytrigger an alarm or other response 76 (e.g., taking a snapshot).

[0077] An embodiment of an exemplary technique for performing analysisand detecting tripwire events is shown in FIG. 11. FG objects are firstdetermined from the video using object segmentation 740. Objectsegmentation 740 may, for example, comprise steps 7221, 7222 and 7223shown in FIG. 10 and discussed above. The location of a FG object isthen tested 741 to determine if it overlaps a line representing atripwire 742 As discussed above, in an exemplary embodiment in whichtripwire lines are always assumed to be on the ground, an object isdetermined to cross a tripwire if the bottom portion of the objectoverlaps a tripwire line. If it is determined that no overlap occurs,there is no tripwire event 743. If there is overlap, then, if onlycrossings in a specified direction are considered to be tripwire events,the direction of crossing is tested 744, and those crossings notoccurring in the specified direction are not considered to be tripwireevents 745. If crossings in either direction represent tripwire events,then the process skips the test of step 744. If step 744 has beenperformed and yields a positive result, or if step 744 is not performed,then additional inquiries 746 may also be performed. Such inquiriesmight, for example, include determining a particular characteristic ofan object of concern (e.g., a car, a truck, a blue car, a blue stationwagon, a car smaller than a certain size, etc.) or a particular object(e.g., a particular person's face, a license plate number, etc.). Ifsuch inquiries 746 return positive results, or if no such inquiries aremade, then the process determines that a tripwire event has occurred747. Of course, should such inquiries 746 be made and return negativeresults, then it would be determined that no tripwire event hasoccurred.

[0078] Several methods for implementing the determination of directionof a crossing 744 are possible. As a first example, it may beimplemented through the use of optical flow methods to objects detectedas crossing the tripwire; the use of optical flow methods could alsoserve to obviate the need for object segmentation. As a second example,trajectory information may be used from object tracking (in step 7222 ofFIG. 10). As a third example, it may be implemented by setting upsecondary (dummy) tripwires on either side of each actual tripwireentered by the user and determining in what order the secondarytripwires are crossed when the actual tripwire is crossed.

[0079] Calibration 72 is of particular importance in the execution ofstep 74, particularly if only a particular type of object is ofinterest. For example, if people are the objects of interest,calibration 72 permits step 74 to discriminate between, for example,people and objects that are either smaller (e.g., cats and mice) orlarger (e.g., groups of people and cars) than people.

[0080] When data has been gathered, it can then be reported to a user75. In an exemplary embodiment of the invention, a user can query thesystem for results using a graphical user interface (GUI). In thisembodiment, summary information and/or detailed data on one or moreindividual detections may be displayed. Summary information may includeone or more of the following: number of detections, number of people (orother objects of interest) detected, number of multi-person(multi-object) detections (i.e., when multiple persons (or other objectsof interest) cross simultaneously), number of people (objects) crossingin each direction, any or all of the preceding within a user-selectedtime window, and one or more time histograms of any or all of thepreceding. Details on a single detection may include one or more of thefollowing: time, direction, number of people (objects) crossing, size ofobject(s) crossing, and one or more snapshots or videos taken around thetime of the detection.

[0081]FIGS. 12 and 13 show sample screen shots of illustrative reportingdisplays, in an exemplary embodiment. FIG. 12 shows summary information121 about crossings of a tripwire 124 spanning a corridor. In thisparticular illustration, the screen shows live video 123 of the areaincluding tripwire 124. Also included is a caption 125 giving the periodduring which surveillance has been occurring (i.e., a time window) andduring which crossing events have been logged. Summary information 121includes numbers of crossings and their directions. In this case, theuser has further specified that the specific crossing times and datesshould be displayed 122.

[0082]FIG. 13 shows individual information about particular crossingevents; these crossing events happen to correspond to the specificcrossing times and dates 122 in FIG. 12. In the display of FIG. 13, theuser has chosen to display a snapshot of each crossing event, along withits time and date. In particular, the snapshots 131 and 132 correspondto crossing events in the area shown in video 123 of FIG. 12. In afurther embodiment, the user may be able to click on a snapshot or abutton associated with a snapshot to view a corresponding video takenaround the time of the crossing event.

[0083] An example of an application of the inventive video tripwire isthe detection of “tailgating.” Tailgating describes an event in which acertain number of people (often one person) is permitted to enter anarea (or the like) and one or more others try to follow closely to alsogain entry. FIG. 14 depicts a flowchart of a method for implementing atailgating detection system. In this embodiment, it is assumed that avideo surveillance camera is installed in such a position as to be ableto record entries through an entrance, for example, a door or aturnstile. Furthermore, the camera has to be calibrated as discussedabove. The system begins by detecting that a person is entering, or isabout to enter, through the entrance 141. This may be accomplished byany number of means; for example, one may have to enter money, enter acode on a keypad, or swipe a card through a card reader, or the systemmay use a video-based detection method to visually detect the opening ofthe entrance (this would have the advantage of not requiring aninterface with external equipment (card reader, keypad, or the like),which may make for easier installation and implementation in someenvironments). When an entry is detected, surveillance begins 142.During this surveillance, the system detects objects moving through theentrance and analyzes them to determine how many people have entered.This may involve face detection, as mentioned above, if the camera issituated so as to be able to record faces. The system then determineswhether the number of people who entered is permissible 143. In theillustrated embodiment, only one person is permitted to enter at a time;however, in more general embodiments this could be any selected number.If one person (the permissible number) enters, then no response isnecessary 144. On the other hand, should more than one person (more thanthe permissible number) enter, a response is triggered 145. Such aresponse may include, for example, sounding an alarm, taking a snapshot,or recording video surrounding the entry. An added advantage of a systemusing either of the latter two responses is that it may provide usefulevidence in tracking down a person using a stolen card, in the case of asystem with a card reader.

[0084] The embodiments illustrated and discussed in this specificationare intended only to teach those skilled in the art the best way knownto the inventors to make and use the invention. Nothing in thisspecification should be considered as limiting the scope of the presentinvention. The above-described embodiments of the invention may bemodified or varied, and elements added or omitted, without departingfrom the invention, as appreciated by those skilled in the art in lightof the above teachings. It is therefore to be understood that, withinthe scope of the claims and their equivalents, the invention may bepracticed otherwise than as specifically described.

What is claimed is:
 1. A video tripwire system comprising: a sensingdevice producing video output; and a computer system, including a userinterface, for performing calibration and for gathering and processingdata based on video output received from the sensing device, the userinterface comprising input means and output means, said input meansincluding a graphical user interface, wherein the computer systemdisplays processed data, and wherein said graphical user interface isadapted to enable a user to draw a video tripwire on at least one of avideo image of said video output or a snapshot taken from said videooutput.
 2. The video tripwire system according to. Claim 1, wherein saidgraphical user interface comprises a point-and-click interface.
 3. Thevideo tripwire system according to claim 2, wherein the point-and-clickinterface comprises at least one menu.
 4. The video tripwire systemaccording to claim 1, wherein said graphical user interface is furtheradapted to enable a user to enter at least one tripwire rule.
 5. Thevideo tripwire system according to claim 4, wherein said at least onetripwire rule includes at least one of directionality, object type, andobject speed.
 6. The video tripwire system according to claim 4, whereinsaid graphical user interface includes at least one menu adapted toenable a user to enter said at least one tripwire rule.
 7. A videotripwire user interface comprising: a graphical user interface adaptedto enable a user to draw a video tripwire on at least one of a videoimage and a snapshot from a video stream.
 8. The user interfaceaccording to claim 7, wherein the graphical user interface comprises apoint-and-click interface.
 9. The user interface according to claim 7,wherein the graphical user interface comprises at least one menu. 10.The user interface according to claim 7, wherein the graphical userinterface is further adapted to enable a user to enter at least onetripwire rule.
 11. The user interface according to claim 7, wherein thegraphical user interface is further adapted to enable a user to enter atleast one video even rule.
 12. A method of initializing a video tripwiresystem, comprising: entering parameters; and drawing a video tripwire ona least one of a video image and a snapshot from a video stream.
 13. Themethod according to claim 12, wherein said drawing a video tripwirecomprises: using a graphical user interface.
 14. The method according toclaim 13, wherein the graphical user interface comprises apoint-and-click interface.
 15. The method according to claim 12, furthercomprising: entering at least one tripwire rule.
 16. The methodaccording to claim 15, wherein the tripwire rule includes at least oneof directionality, object type, and object speed.
 17. The methodaccording to claim 15, wherein said entering comprises: using apoint-and-click interface.
 18. The method according to claim 17, whereinthe point-and-click interface comprises at least one menu.
 19. Themethod according to claim 12, further comprising: entering at least onevideo event rule.
 20. A computer-readable medium containing instructionsthat, when executed by a computer, cause the computer to be adapted toenable a user to perform the method of claim
 12. 21. Thecomputer-readable medium according to claim 20, further containinginstructions that, when executed by a computer, cause the computer tocreate a graphical user interface adapted to enable a user to performsaid drawing.
 22. The computer-readable medium according to claim 21,wherein the graphical user interface comprises a point-and-clickinterface.
 23. The computer-readable medium according to claim 22,wherein the point-and-click interface includes at least one menu.
 24. Avideo tripwire system comprising: a computer system; and thecomputer-readable medium according to claim
 20. 25. A video tripwiresystem comprising: a sensing device producing video output; and acomputer system, including a user interface, for performing calibrationand for gathering and processing data based on video output receivedfrom the sensing device, the user interface comprising input means andoutput means, wherein the computer system displays processed data, andwherein the computer system includes software permitting a user to enterat least one virtual tripwire, the video tripwire comprising at leastone of a curved video tripwire, a multi-segment video tripwire, and amultiple parallel video tripwire.
 26. A method of implementing a videotripwire system comprising: calibrating a sensing device to determinesensing device parameters for use by the system; initializing thesystem, including entering at least one virtual tripwire, said at leastone virtual tripwire comprising at least one of a curved tripwire, amulti-segment tripwire, and a multiple parallel tripwire; obtaining datafrom the sensing device; analyzing the data obtained from the sensingdevice to determine if the at least one virtual tripwire has beencrossed; and triggering a response to a virtual tripwire crossing.
 27. Avideo tripwire system comprising: a sensing device providing outputdata; and a computer system receiving the output data and comprising: auser interface; at least one processor; and a computer-readable mediumcontaining software implementing the method of claim 26.